Spatial/joint space-time motion segmentation of image sequences by level set pursuit

Segmentation of image sequences based on motion is an important problem with numerous applications in video processing (objectbased frame conversion) and video compression (MPEG-4) [1]. Motion-based segmentation can be carried out either on a frame by frame basis, resulting in a 2D spatial region segmentation in each frame of the sequence, or it can be carried out on a multi-frame basis, whereby the image sequence is treated as a function defined on a spatio-temporal volume, resulting in a spatio-temporal 3D spatio-temporal volume segmentation problem [2]. The basic problem we address in this paper is the following: Given a set of sparse point correspondences, how does one obtain a dense motion field and a motion-based segmentation of the image sequence ? We propose a novel solution to this problem by formulating it as pursuit in segmentation space. This segmentation is defined by level set evolution equations, allowing changes in segmentation topology [4]. The main novelty of our proposed algorithm is that the number of distinct motion regions and their parameters need not be known prior to segmentation and are determined by the segmentation itself. Furthermore, the algorithm we propose applies equally to (frame-by-frame) spatial motion-based segmentation and to (multiframe) joint space-time motion-based segmentation. This extends our prior work on motion-based image segmentation with level sets in both the spatial [3] and spatio-temporal domains [2] where the number of distinct motion regions as well as their precise motion parameters need to be computed through a complex clustering operation prior to segmentation. We formulate motion-based image segmentation as pursuit in the space of image segmentations, in analogy to classical matching pursuit, the basic idea consisting of iteratively segmenting the image by focusing on residual regions. The segmentation obtained by our algorithm is based on motion alone and does not use intensity boundaries as an auxiliary. We formulate our algorithm for the case of spatial motion segmentation and illustrate it on a real image sequence with natural motion; our proposed algorithm applies verbatim to the case of joint space-time motion segmentation with level sets, and we refer the reader to [2] for details of the latter.